Skip to content

Expose stage/release status in schema#5443

Draft
janniklasrose wants to merge 11 commits into
mainfrom
janniklasrose/sdk-field-status-in-schema
Draft

Expose stage/release status in schema#5443
janniklasrose wants to merge 11 commits into
mainfrom
janniklasrose/sdk-field-status-in-schema

Conversation

@janniklasrose

@janniklasrose janniklasrose commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Stacked on #5484

Changes

Expose launch stage/release status in the bundle schema

Why

Non-GA fields should be annotated as such when viewing a bundle in IDEs.

Tests

Example vector search endpoint:

Enum with one GA, one PuPr:
Screenshot 2026-06-04 at 14 29 40

PuPr field without any annotation:
Screenshot 2026-06-04 at 14 29 49

Comment thread bundle/docsgen/main.go Outdated
@@ -139,13 +139,51 @@ func assignAnnotation(s *jsonschema.Schema, a annotation.Descriptor) {
s.Deprecated = true
s.DeprecationMessage = a.DeprecationMessage
}
if a.Preview == "PRIVATE" {
s.DoNotSuggest = true
if a.Preview != "" {

@ilyakuz-db ilyakuz-db Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shreyas-goenka do we still use docsgen?

@janniklasrose janniklasrose marked this pull request as ready for review June 8, 2026 11:25
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Approval status: pending

/bundle/ - needs approval

14 files changed
Suggested: @pietern
Also eligible: @shreyas-goenka, @andrewnester, @denik, @lennartkats-db, @anton-107

General files (require maintainer)

66 files changed
Based on git history:

  • @pietern -- recent work in bundle/internal/schema/, python/databricks/bundles/pipelines/_models/, bundle/schema/

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Commit: e6b64cd

Run: 27259369137

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 7 15 261 926 6:41
💚​ aws windows 7 15 263 924 13:08
💚​ aws-ucws linux 7 15 357 840 6:48
💚​ aws-ucws windows 7 15 359 838 11:52
💚​ azure linux 1 17 264 924 6:14
💚​ azure windows 1 17 266 922 10:57
💚​ azure-ucws linux 1 17 362 836 10:06
💚​ azure-ucws windows 1 17 364 834 11:58
💚​ gcp linux 1 17 260 927 7:41
💚​ gcp windows 1 17 262 925 11:25
22 interesting tests: 15 SKIP, 7 RECOVERED
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/replace_existing 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_projects/update_display_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/grants/select 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
Top 31 slowest tests (at least 2 minutes):
duration env testname
6:15 aws windows TestAccept
6:08 gcp windows TestAccept
6:01 aws-ucws windows TestAccept
5:57 azure windows TestAccept
5:50 azure-ucws windows TestAccept
4:32 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:26 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:19 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:17 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:35 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:24 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:24 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:18 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:12 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:08 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:00 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:58 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:56 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:53 gcp linux TestAccept
2:53 aws linux TestAccept
2:52 azure-ucws linux TestAccept
2:51 aws-ucws linux TestAccept
2:50 azure linux TestAccept
2:46 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:45 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:45 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:43 azure-ucws linux TestSQLExecScalar
2:37 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:36 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:30 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:29 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

@janniklasrose janniklasrose enabled auto-merge June 8, 2026 12:35
pietern added 8 commits June 9, 2026 16:26
Generate the workspace and account command stubs from a checked-in
.codegen/cli.json instead of reflecting over an external API spec, so
codegen is reproducible from data committed to this repo.

- internal/cligen: a self-contained, data-driven generator (model,
  templates, name/casing derivation) that renders the command stubs
  from cli.json
- bundle/internal/schema: read API annotations (descriptions, enums,
  field behaviors) directly from cli.json; drop the OpenAPI pseudo-spec
  reconstruction and the raw-spec environment path
- Taskfile: wire the cli.json-based generators into `task generate`
- regenerate the committed codegen artifacts from cli.json

Co-authored-by: Isaac
splitASCII drives every casing function and is a faithful port of genkit's
Named.splitASCII, whose nearest-letter scanning produces several non-obvious
splits (acronym head/tail handling, digits never starting a word). Pin those
as a curated table so the port can't drift.

Co-authored-by: Isaac
Pure move of the name-splitting engine (splitASCII and its
condAtNearestLetters/searchNearest helpers) out of names.go so it sits
next to split_test.go. The casing functions stay in names.go.

Co-authored-by: Isaac
Add table cases for the special-cased early returns in camelName/snakeName
and the '$'-skip branch in splitASCII, bringing the pure name/casing logic
to full statement coverage.

Co-authored-by: Isaac
Switch the CLI to genkit's native cli_v1 producer (`genkit update-sdk`,
mode cli_v1). The cli.json schema keys/refs now use the Go SDK's
package.PascalName rendering and the file carries every message —
including request/response wrappers, which are inert for the CLI since it
reads only the schemas its commands reference.

- derive DoNotSuggest from launch_stage == PRIVATE_PREVIEW instead of the
  dropped preview flag (cli_json.go, parser.go), sourcing the signal from
  the single launch-stage source of truth
- Taskfile generate-cli-json: build genkit from universe HEAD and run
  update-sdk, since the cli_v1 producer is decoupled from the spec
- regenerate annotations, jsonschema (+ for-docs), and docs; compute.Kind's
  description now resolves via the package.PascalName key
- cli_json_test: assert the spec sha via the _openapi_sha file (genkit no
  longer writes it into cli.json metadata)

Co-authored-by: Isaac
The cli_v1 producer emits go_v0 package.PascalName schema keys that match
the Go SDK type names directly (e.g. serving.OpenAiConfig), so the
Go-name-to-spec-name fixup is never reached. Regenerating the annotations,
json schemas, and docs is byte-identical with it removed.

Co-authored-by: Isaac
- drop the .codegen/cli.json entry from the test-task sources; cli.json
  changes are already covered by the dedicated cli.json/cligen tasks
- restore the "Generating CLI code..." echo to match origin/main

Co-authored-by: Isaac
@janniklasrose janniklasrose force-pushed the janniklasrose/sdk-field-status-in-schema branch from a4089fd to 1a4d7eb Compare June 9, 2026 23:03
@janniklasrose janniklasrose changed the base branch from main to cligen June 9, 2026 23:04
auto-merge was automatically disabled June 9, 2026 23:04

Merge commits are not allowed on this repository

Comment thread bundle/docsgen/main.go Outdated
Comment thread bundle/internal/annotation/descriptor.go Outdated
Comment thread bundle/internal/schema/annotations.go Outdated
Comment on lines +224 to +229
case "PRIVATE_PREVIEW":
return "[PrPr]"
case "PUBLIC_BETA":
return "[Beta]"
case "PUBLIC_PREVIEW":
return "[PuPr]"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PrPr and PuPr aren't used in Databricks public documentation, but for space-constrained values I think it's still more appropriate. wdyt @juliacrawf-db ?

@juliacrawf-db juliacrawf-db Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly I think those are very Databricks-internally-specific acronyms and we shouldn't be putting them in public facing content/APIs/source because the meaning isn't super obvious. Ideally it's PublicPreview and PrivatePreview, but if it's constrained maybe PrivatePr and PublicPr (imo the private/public part seems more important to expand)? (Or do you need it to be even shorter?)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... now I'm having second thoughts about that...because "Pr" likely will get translated as PR in this context...

Comment thread bundle/internal/schema/annotations.go Outdated
Comment thread bundle/internal/schema/annotations.go Outdated
// field's description. cli.json is filtered at min-stage=PRIVATE_PREVIEW
// upstream, so DEVELOPMENT never reaches here and GA yields no tag.
//
// Keep this in sync with previewTag in bundle/docsgen/main.go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why two sources of truth?

Read launch_stage, enum_launch_stages and enum_descriptions from the
checked-in .codegen/cli.json and surface them in the generated bundle
schema, docs and Python models:

- Prefix field descriptions with [Public Preview] / [Beta] /
  [Private Preview] so the stage shows in editor hover tooltips and the
  generated docstrings.
- Emit the parallel enumDescriptions array so VSCode renders a per-value
  stage label and description in autocomplete dropdowns.
- Carry the raw x-databricks-launch-stage alongside the existing
  x-databricks-preview, and hide PRIVATE_PREVIEW fields via doNotSuggest.

PreviewTag lives in bundle/internal/annotation as the single source of
truth shared by the schema and docs generators. The PUBLIC_BETA label is
"[Beta]" to match Databricks public documentation.

cli.json is filtered upstream at min-stage=PRIVATE_PREVIEW, so no
DEVELOPMENT surface reaches the extractor and launch_stage is already
clean (GA plus the three preview stages). The OpenAPI-spec-based
DEVELOPMENT filtering and preview normalization the earlier version
carried are no longer needed.

assignAnnotation flags an open question (in a comment) on whether either
x-databricks-preview or x-databricks-launch-stage is consumed downstream
of the published schema, pending confirmation with the team.

Regenerated jsonschema.json, jsonschema_for_docs.json,
annotations_openapi.yml, the bundle docs and the Python (pydabs) models.

Co-authored-by: Isaac
Map ./bundle/schema/jsonschema.json onto databricks.yml / bundle.yml in
.vscode/settings.json so the schema generated in this repo is the one the
editor validates and autocompletes against.

Co-authored-by: Isaac
@janniklasrose janniklasrose force-pushed the janniklasrose/sdk-field-status-in-schema branch from 1a4d7eb to e69435c Compare June 10, 2026 06:56
Comment thread bundle/internal/schema/annotations.go Outdated

// previewTagShort is the compact counterpart to previewTag, used for per-enum-
// value labels where vertical space in the dropdown is tighter.
func previewTagShort(launchStage string) string {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to annotation.PreviewTagShort

// derivable from it (PRIVATE iff PRIVATE_PREVIEW) and is kept only for
// backwards compatibility, so PRIVATE_PREVIEW must hide the field here too.
//
// OPEN QUESTION (confirm with the team before finalizing this PR): is

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

open question

Addresses review feedback: PreviewTagShort now lives alongside PreviewTag
in bundle/internal/annotation, keeping the launch-stage label vocabulary
in a single place. Pure refactor, no change to generated output.

Co-authored-by: Isaac
@janniklasrose janniklasrose marked this pull request as draft June 10, 2026 14:18
Base automatically changed from cligen to main June 11, 2026 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants